Skip to content

[Feature] Open Telemetry Exporter Addition - #60

Merged
Saaketh0 merged 18 commits into
mainfrom
feature/otel-exporter
Sep 9, 2026
Merged

[Feature] Open Telemetry Exporter Addition#60
Saaketh0 merged 18 commits into
mainfrom
feature/otel-exporter

Conversation

@Saaketh0

Copy link
Copy Markdown
Collaborator

Added the OTel exporter for the codebase, allowing requests to be converted and sent to a OTLP Receiver. Kept old telemetry path for back-compat, will remove later.

Also added basic logic for global_controller.py starting new processes, which will soon extend to the reconciliation, llm_proxy, and polling processes.

Saaketh0 and others added 16 commits August 25, 2026 17:26
generate_docker()/generate_workflow_docker() now recursively sweep every
.py file under the project directory into the build context, preserving
directory structure, so helper files that aren't declared as an agent
entrypoint still make it into the image. Generated dirs (docker_container/,
stubs/, grpc_stubs/) are excluded at the project root only, not at every
depth.

Stub files are placed at their agent's declared entrypoint path (mapped
from global_controller.yaml) instead of a hardcoded guess, so a stub
overwrites the exact real file it replaces. Guards against absolute and
'..'-containing entrypoints, symlinked sources, and symlinked-destination
escapes, with warnings on unsafe or unmapped stubs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…heckpoint)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

# Conflicts:
#	examples/portfolio/agents/advisor_agent.py
Users had no way to get API keys (OpenAI, Anthropic, embedding models)
into an agent container. Add a top-level `env_file` key to
global_controller.yaml pointing at a local .env file, which reaches every
container as `docker run --env-file`.

- resolve_env_file validates the path before anything launches, so a
  missing .env fails at deploy time instead of deep inside a container.
  Relative paths resolve against the project root, matching entrypoint.
- env_file_args is a context manager owning the local-vs-remote decision
  and the cleanup, so both runtimes share one code path. Local containers
  read the original file; remote containers get a copy that is deleted as
  soon as `docker run` returns, whether or not it succeeded.
- GlobalController._push_file streams the file over ssh under `umask 077`
  rather than scp, so the copy is never briefly world-readable and the
  secret never lands in a command line. _run_cmd's ssh options moved to a
  shared _ssh_args.

--env-file is appended after the explicit -e VENTIS_* flags; Docker gives
those precedence regardless of order, so a stray VENTIS_* line in
someone's .env cannot break agent wiring.

Closes #50
Two holes in the remote staging path, both found reviewing the feature
commit.

`umask 077` only governs files the shell creates, and `>` follows
symlinks -- so it did not actually guarantee a 0600 copy. The destination
path is fully predictable (`/tmp/ventis-env-ventis-ec2-<agent>-<n>`), so a
local user on the remote host could pre-create it world-readable, or point
it at a file of their own, and collect the API keys. Remove whatever sits
at the path before writing; `rm -f` unlinks a symlink rather than following
it, so `cat >` then creates a fresh file under the umask.

`_run_cmd` joins its argv with spaces and hands the result to a remote
shell unquoted. `_push_file` quoted its path but the cleanup `rm` did not,
so a container name containing a space split the `rm` into two arguments
that matched nothing -- it exited 0 while the secrets file stayed on the
host, and the returncode check logged nothing. Scrub the name down to
[A-Za-z0-9_.-] in remote_env_path, which also closes the same gap in the
`--env-file` argument and in any future use of that path.

Still open, tracked separately: a push that dies mid-transfer can leave a
copy behind, since the cleanup only covers the `docker run` that follows.
On EC2 the instance is terminated on that path, which disposes of it.
…nup-race fix (pre-pull checkpoint)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This project's agents/workflow import each other's stubs by flat module
name, not by the exporting agent's own entrypoint path. Applying
_stub_destination's entrypoint-mirroring broke both the Workflow
(ModuleNotFoundError: intent_agent) and agent-to-agent calls
(MetricsAgent -> price_agent) on live redeploy. Keeps PR #51's actual
fix (project_dir sweep for unstubbed helper files) intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Metrics/telemetry latency scaled with instance count x per-instance
round-trip time since every instance was polled sequentially, one
blocking the next, with the following tick only starting after the
whole pass finished. Extracted the per-instance body into
_poll_one_instance (whole body wrapped in one top-level try/except,
since ThreadPoolExecutor.map() re-raises on first exception when
results are consumed) and run all instances concurrently via the same
ThreadPoolExecutor pattern _trigger_cleanup already used.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Saaketh0
Saaketh0 requested a review from iidsample August 31, 2026 21:02
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: bdd04bb5-91e1-41ce-b4b6-9483367dbff5


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Resolved conflicts:
- metrics_agent.py: removed duplicate imports
- portfolio_workflow.py: used main's simpler intent handling
- cli.py: kept both comment explanations
- stub_generator.py: used main's version (no entrypoint injection)

otel:
destinations:
- name: railway

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is plan for railway ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write a few comments here what this DB is about

Comment thread ventis/controller/utils/env_file.py Outdated

The user points `env_file` in `config/global_controller.yaml` at a local
`.env` file. Containers on this machine read that file directly; containers
on a remote host get a short-lived 0600 copy. Either way the file reaches

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is input the env_file ?
How do we specify different environment files

)

# Start background cleanup thread
self._cleanup_ready = threading.Event()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of this event ?

Resolves two conflicts:
- global_controller.py: import-ordering only, kept both branches' imports
- env_file.py (add/add): took main's version, a strict superset that adds
  platform_secrets_file()/DEFAULT_SECRETS_FILE for managed deployments on
  top of the same resolve_env_file(config, base_dir) public API this
  branch already calls unchanged

Also fixes a pre-existing SyntaxError on this branch's HEAD (unrelated to
the merge): cmd_build()'s generate_workflow_docker() and generate_docker()
calls each repeated project_dir= and stub_entrypoints= as duplicate
keyword arguments, a leftover from an earlier merge. ventis/cli.py could
not even be imported/compiled before this fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Saaketh0
Saaketh0 merged commit 28c9cbe into main Sep 9, 2026
4 of 5 checks passed
Saaketh0 added a commit that referenced this pull request Sep 9, 2026
Resolves two conflicts:
- global_controller.py: _poll_controllers() conflicted with main's
  sequential loop because this branch parallelized it (ThreadPoolExecutor
  + a new _poll_one_instance() method). Diffed both versions ignoring
  whitespace and confirmed every line of actual per-instance logic
  (OTEL write_waiting_rows, send_runtime_information, metrics, status)
  is byte-identical between branches -- this branch's version is a
  strict superset (same logic, parallelized), so kept it wholesale and
  left every other line in the file matching main exactly.
- env_file.py (add/add): same conflict as PR #60, same fix -- took
  main's version, a superset that adds platform_secrets_file() on top
  of the unchanged resolve_env_file(config, base_dir) API this branch
  already calls.

Also fixes the same pre-existing SyntaxError found on feature/otel-exporter
(shared ancestor): cmd_build()'s generate_workflow_docker() and
generate_docker() calls each repeated project_dir= and stub_entrypoints=
as duplicate keyword arguments. ventis/cli.py could not be imported/
compiled before this fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Saaketh0 added a commit that referenced this pull request Sep 9, 2026
fixes/ventis-cli-fixes and feature/config-reloading are parallel lines off
a common ancestor far back (75d7705), not a linear descendant relationship
-- each independently re-implemented overlapping OTel/otel-exporter work
after diverging. Resolved per-file based on which side is the actual
superset, verified with full-file diffs against both parents, not a
blanket "one side wins" rule:

- env_file.py (add/add): took feature/config-reloading's side -- it already
  has the managed-secrets platform_secrets_file()/DEFAULT_SECRETS_FILE
  support merged in from main (via this session's PR #60/#61 work), which
  fixes/ventis-cli-fixes predates and lacks entirely.
- db.py, convert.py, test_otel_exporter_fields.py (add/add): took
  fixes/ventis-cli-fixes -- cost-lookup try/except fallback and 64-bit
  Future.id/span_id, both strict additions config-reloading never touched.
- _runtime.py: took fixes/ventis-cli-fixes -- adds a port-conflict retry
  loop around docker run; verified the whole-file diff is confined to
  this one function, and that registering the Redis endpoint AFTER the
  retry loop (instead of before, like config-reloading) is required once
  host_port can change mid-retry, not just a style choice.
- global_controller.py: took fixes/ventis-cli-fixes -- verified via full
  diff (not just the 5 conflict markers) that every other difference in
  the file, conflicted or already auto-merged, also favors this side
  (.car-layout project root resolution, persisted dashed-uuid project_id,
  VENTIS_REDIS_HOST env override for a containerized GC's own Redis
  connection).
- cli.py, test_cli.py: took fixes/ventis-cli-fixes -- folds build into
  deploy per this PR's own description, fixes entrypoint-only stub
  placement, and incidentally fixes the recurring duplicate-keyword
  SyntaxError (project_dir=/stub_entrypoints= repeated) that
  config-reloading's copy still carries since it branched before that got
  fixed elsewhere this session.
- otel_exporter.py (add/add): user decision -- kept fixes/ventis-cli-fixes's
  RedisClient(host="host.docker.internal") as-is, despite it not matching
  the VENTIS_REDIS_HOST env-var pattern this same PR introduces for
  GlobalController's own Redis connection a few lines away in
  global_controller.py. Flagged as a known inconsistency, not fixed here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants